Covariance of Motion and Appearance Featuresfor Spatio Temporal Recognition Tasks

In this paper, we introduce a novel descriptor for employing covariance of motion and appearance features for human action and gesture recognition. In our approach, we compute kinematic features from optical flow and first and second-order derivatives of intensities to represent motion and appearance respectively. These features are then used to construct covariance matrices which capture joint statistics of both low-level motion and appearance features extracted from a video. Using an overcomplete dictionary of the covariance based descriptors built from labeled training samples, we formulate human action recognition as a sparse linear approximation problem. Within this, we pose the sparse decomposition of a covariance matrix, which also conforms to the space of semi-positive definite matrices, as a determinant maximization problem. Also since covariance matrices lie on non-linear Riemannian manifolds, we compare our former approach with a sparse linear approximation alternative that is suitable for equivalent vector spaces of covariance matrices. This is done by searching for the best projection of the query data on a dictionary using an Orthogonal Matching pursuit algorithm. We show the applicability of our video descriptor in two different application domains namely human action recognition and gesture recognition using one shot learning. Our experiments provide promising insights in large scale video analysis.